Prediction of Dynamical Systems by Recurrent Neural Networks

Recurrent neural networks in general achieve better results in prediction of time series then feedforward networks. Echo state neural networks seem to be one alternative to them. I have shown on the task of text correction, that they achieve slightly better results compared to already known method based on Markov model. The major part of this work is focused on alternatives to recurrent neural networks training that are based on Kalman filtration modifications. I describe in detail the training by filters: Extended Kalman Filter, Unscented Kalman Filter (UKF), nprKF Filter and their joint versions UKFj and nprKFj. Contribution of this work is presentation of simpler equations for individual filters, because they are modified specifically for recurrent neural network training. Filter UKFj in context of recurrent neural networks was probably firstly described in my work. I compare individual filters with each other and also with gradient descent method Truncated Backpropagation Through Time (BPTT(h)). I show the results are consistently better when comparing recurrent neural networks trained by these advanced methods with BPTT(h). In the like manner, Extended Kalman Filter achieves worse results compared to the other filters, which on the other hand achieve comparable results with each other. I describe how to speed up their computation by utilizing the graphics card. My work is one of the first (if not the first) that focuses on recurrent neural network training utilizing the processor on graphics card. This paper represents my dissertation summary.